Skip to content

UFAL/Add CCMM 1.1.0 OAI-PMH crosswalk (ccmm-xml metadataPrefix) - #1262

Open
milanmajchrak wants to merge 9 commits into
dtq-devfrom
ufal/oai-pmh-new-crosswalk
Open

UFAL/Add CCMM 1.1.0 OAI-PMH crosswalk (ccmm-xml metadataPrefix)#1262
milanmajchrak wants to merge 9 commits into
dtq-devfrom
ufal/oai-pmh-new-crosswalk

Conversation

@milanmajchrak

@milanmajchrak milanmajchrak commented Feb 27, 2026

Copy link
Copy Markdown
Collaborator

Problem description

Add CCMM 1.1.0 OAI-PMH crosswalk — exports metadata as CCMM 1.1.0 over OAI-PMH (metadataPrefix: ccmm-xml), required for NMD/NRP.

New ccmm.xsl, registered in xoai.xml (Default + openaire_data context), plus unit tests and test data.

Analysis

Run over all 2 628 live LINDAT records and validated against CCMM 1.1.0 with XSD 1.1:

before after
records that transform 2 203 2 628
ListRecords pages a harvester can fetch 0 / 32 32 / 32
XSD 1.1 valid 2 203 2 628
records making a false statement 1 216+ 0
DSpace metadata paths read 29 / 89 77 / 89
mutants caught by the test suite 20 / 72 45 / 46
distinct IRIs returning HTTP 404 16 0

What the feed was getting wrong

  1. Transform errordoc:field[@name='value'] is a sequence; the 425 records carrying two language variants of a field threw XPTY0004. OAIException becomes HTTP 500 for the whole page, so every one of the 32 pages failed and nothing was harvestable.

  2. Access rights were wrong on 1 216 of 2 628 records. They came from others/access-status, which DefaultAccessStatusHelper derives from the READ policies of the primary bitstream — on a CLARIN repository that does not say whether the resource can be downloaded. The gate is the licence category in dc.rights.label (PUB/ACA/RES) together with others/restrictedAccess. Verified against the live repository with anonymous ranged GETs: 12/12 sampled records published as "restricted access" return HTTP 206, 11/12 published as "open access" return 401. Those two signals now decide first and access-status is the fallback.

  3. Vocabularies — every model.ccmm.cz/vocabulary/… IRI returned 404. Replaced with the registers CCMM names in en/dsv.ttl. Every emitted IRI and label was then checked against the live register, which answers a real concept with its prefLabel and a bogus one with nothing. That found three more errors: all four date_type labels were the concept id (Issued) rather than the register's prefLabel (Date Issued); conforms_to_standard identified the profile by this stylesheet's own XML namespace under the label "CCMM 1.1", a string absent from the whole CCMM release; and RelationType/Replaces / IsReplacedBy do not exist — the concepts are Obsoletes and IsObsoletedBy.

  4. Licencelicense/label carried the CLARIN access category (PUB/ACA/RES) on 1 932 records instead of the licence name, which was demoted to a description. A free-text dc.rights.uri was copied verbatim into license/iri.

  5. Datespublication_year took min() over the approximate years even when the record stated an exact issue date (48 records backdated by up to 39 years); a comma enumeration (1920, 1932) became a continuous interval although the source names two discrete years; dc.date.issued = "0000" fell through to the DSpace ingest timestamp; and Created reused the deposit year on 158 records that state a creation year in local.additional.metadata.

  6. Agents — the type was hard-coded per source field, so film studios, institutions and ANVL placeholders were published as prov#Person. Names are now typed from the value, packed values are split, placeholders produce no agent at all, and given_name/family_name are emitted for the 15 448 values that carry the split.

  7. Subjects — hierarchical vocabulary paths (People::Masaryk …) were published verbatim as titles. The leaf is now the title and the whole path a classification_code.

Fields that were being dropped

The contact person and the repository contact, the owning collection, dc.identifier.other, dc.source.uri, local.demo.uri, every dc.relation qualifier, dc.format, dc.coverage.placeName, language names, local.sponsor, local.additional.metadata, local.size.info, local.refbox.format and the whole metashare tree. Bitstreams of the ORIGINAL bundle now become distribution entries with format, byte size and checksum.

Portability

None of this changes LINDAT output, but the crosswalk ships in DSpace core: ISO 639-1 and region-tagged language codes are accepted (en, en_US, de-DE), the stock DSpace type list is mapped, every date is proved castable before it is emitted, the funding guard names the whole grantAgreement prefix instead of any info: URI, and a repository with no url is no longer described as living at each item's own address.

Tests

70 methods over 9 fixtures, including a stock-DSpace record, every date shape DSpace stores, and both CLARIN access shapes. Element order is asserted directly with preceding-sibling — the CCMM schema is an xs:sequence, and validating against it would pull xml.xsd and GML over the network at build time — plus closed-set guards so no off-register string or unresolvable IRI host can come back. Mutation testing over 46 single-edit mutants: 45 caught, against 20 of 72 before. The one survivor edits a FormatDate branch that its callers make unreachable.

Problems

CCMM asks for at least one subject from the FRASCATI FORD vocabulary. LINDAT has no FORD field, so no record satisfies that yet — it needs a submission-form change, not a crosswalk change.

Not mapped, because none of it describes the resource: local.branding, local.hidden, local.hasCMDI, the local.files.* counters, the deposit licence blob, and bundles other than ORIGINAL. Carried in the source but with nowhere to go in CCMM 1.1.0: the per-file description on a bitstream, since distribution_downloadable_file has no description element.

Resolves: ufal#1145

Manual Testing (if applicable)

Copilot review

  • Requested review from Copilot

🤖 Generated with Claude Code

Copilot AI review requested due to automatic review settings February 27, 2026 12:48

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR implements a new OAI-PMH crosswalk for exporting DSpace metadata in CCMM (Czech Common Metadata Model) 1.1.0 format, required for NMD/NRP (National Repository of Gray Literature) compliance in the Czech Republic. The implementation follows the standard DSpace OAI-PMH crosswalk pattern used by other metadata formats in the repository.

Changes:

  • New XSL transformation stylesheet mapping DSpace internal metadata to CCMM 1.1.0 XML schema
  • Configuration updates to register the new crosswalk in both Default Context and openaire_data Context
  • Comprehensive test suite with 14 unit tests and dedicated test data file

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 14 comments.

File Description
dspace/config/crosswalks/oai/xoai.xml Registers ccmm format (metadataPrefix: ccmm-xml) in Default and openaire_data contexts
dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl 633-line XSLT 2.0 stylesheet implementing DSpace to CCMM metadata transformation with fallback handling for required fields
dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java Test class with 14 tests validating core CCMM elements (identifiers, creators, subjects, license, etc.)
dspace-oai/src/test/resources/xoai-ccmm-test.xml Test fixture simulating typical NMD/NRP dataset metadata with Czech academic repository data

Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java Outdated
- Extract hardcoded fallback strings as XSLT variables at top of stylesheet
  (Unknown Repository, http://unknown.repository, Untitled, unspecified, 9999)
- Add detailed comment explaining Handle URL extraction assumptions
- Add comment explaining publisher to Distributor role mapping rationale
- Fix time_reference fallback: use dc.date.accessioned instead of hardcoded
  9999-01-01 when issued/available dates are missing
- Improve FormatDate fallback: validate first 4 chars are digits before using
  as year, output empty ccmm:date element for invalid input
- Improve identifier fallback logic: check if Handle-pattern URIs were actually
  found rather than just checking for raw element presence
- Add 6 new fallback scenario tests (title, publication_year, subject,
  identifier, repository name, license) with dedicated minimal test fixture
- All 21 tests pass (20 CCMM + 1 QDC)

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 10 comments.

Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/xoai.xml
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Verified against all 2628 live LINDAT records; every output validates
against CCMM 1.1.0 in XSD 1.1 (0 invalid).

- fix XPTY0004: doc:field[@name='value'] returns a sequence, and records
  with two language variants of a field passed it to concat()/contains().
  425 of 2628 records aborted; the resulting OAIException becomes HTTP 500
  for the whole ListRecords page, so all 32 pages failed and nothing was
  harvestable
- replace the model.ccmm.cz IRIs, which all return 404, with the registers
  CCMM 1.1.0 names in its own specification (en/dsv.ttl scope notes):
  COAR resource types and access rights, vocabs.ccmm.cz AgentRole /
  TimeReference / DescriptionType, EU Publications Office for languages
- dc.publisher now maps to AgentRole/Publisher; Distributor does not exist
  at that level of the codelist
- derive access_rights from others/access-status, which DSpace computes in
  DefaultAccessStatusHelper from the actual READ policies. The previous
  branches were dead (local.embargo.termslift occurs 0 times) or inverted
  (others/@restrictedaccess is set on 24 records, all of them open access).
  Correct for 746 of 2628 records before, 2628 of 2628 now
- handle the CLARIN "0000" unknown-date convention and
  local.approximateDate.issued the way ClarinDateService does in the UI:
  the local field replaces dc.date.issued, a range becomes a time_interval
  and the original string is kept verbatim in ccmm:date_information
- emit exactly one Issued time reference plus a Created one, as the CCMM
  profile requires; publication_year and the Issued year now always agree
- stop emitting an invented licence IRI; fall back to others/cc and
  otherwise leave ccmm:license empty and move the wording to
  terms_of_use/description
- the minimal test fixture used to produce schema-invalid output (no
  time_reference at all); it no longer can

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.

Suppressed comments (3)

dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl:174

  • original_repository is populated from the item identifier, so the common hdl.handle.net/... form makes every record identify its item resolver as the repository. XOAI already supplies the actual repository URL at repository/url (ItemUtils.java:347); use that value instead.
                        <xsl:when test="doc:metadata/doc:element[@name='dc']/doc:element[@name='identifier']/doc:element[@name='uri']/doc:element/doc:field[@name='value']">
                            <xsl:variable name="uri" select="(doc:metadata/doc:element[@name='dc']/doc:element[@name='identifier']/doc:element[@name='uri']/doc:element/doc:field[@name='value'])[1]"/>

dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java:125

  • This assertion exceeds the enforced 120-character line limit (checkstyle.xml:47-52). Wrap the expected organization name.
            equalTo("Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics"))));

dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java:182

  • This XPath exceeds the repository's enforced 120-character Java line limit (checkstyle.xml:47-52). Split it into concatenated segments.
            "//ccmm:dataset/ccmm:metadata_identification/ccmm:qualified_relation/ccmm:relation/ccmm:organization/ccmm:name",

Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl Outdated
Comment thread dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java Outdated
Comment thread dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java Outdated
milanmajchrak and others added 6 commits August 28, 2026 12:36
- original_repository/iri named the item itself on every one of the 2628
  records: it was derived with substring-before(uri, '/handle/'), which
  does not match LINDAT's hdl.handle.net URIs, so the otherwise branch
  emitted the whole item URI. XOAI already carries repository/@url; use it.
- xml:lang was hardcoded "en" on subjects, alternate titles, descriptions
  and the rights wording. XOAI nests every value inside an element named
  after its language qualifier, so take the tag from there and fall back
  to "en" only for the unqualified "none" wrapper.

Both covered by new tests. Re-checked over all 2628 live LINDAT records:
still 2628/2628 valid against CCMM 1.1.0 in XSD 1.1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ields it dropped

An adversarial audit over all 2628 live LINDAT records found 30 defects. They fall into three
groups, and the first is the one that matters: the feed was stating things the source does not
support.

False statements, now fixed:
- access_rights was taken from others/access-status alone. DefaultAccessStatusHelper derives that
  from the primary bitstream's READ policies, which on a CLARIN repository does not say whether
  the resource can be downloaded: the gate is the licence category in dc.rights.label plus
  others/restrictedAccess. 1216 of 2628 records carried a wrong statement - 1161 anonymously
  downloadable datasets published as "restricted access", 55 login-gated ones as "open access".
  Verified against the live repository: 12/12 sampled "restricted" records return HTTP 206 to an
  anonymous ranged GET, 11/12 "open" ACA/RES records return 401.
- Every date_type label was the register's concept id ("Issued") rather than its prefLabel
  ("Date Issued") - 10512 labels, none correct.
- conforms_to_standard identified the profile by this stylesheet's own XML namespace under the
  label "CCMM 1.1", a string that appears nowhere in the CCMM release.
- license/label carried the CLARIN access category (PUB/ACA/RES) instead of the licence name,
  which was demoted to a description. 1932 records.
- Hierarchical vocabulary paths ("People::Masaryk ...") were published verbatim as subject titles;
  the leaf is now the title and the path a classification_code.
- Agent type was hard-coded per source field, so film studios and ANVL placeholders were published
  as people. Names are now typed from the value, and placeholders produce no agent at all.
- publication_year took min() over the approximate years even when the record stated an exact
  issue date; a comma enumeration ("1920, 1932") became a continuous interval; a "0000" issue date
  fell through to the ingest timestamp; Created ignored the creation year the record states.
- RelationType/Replaces and IsReplacedBy do not exist in the register - the concepts are
  Obsoletes and IsObsoletedBy. Every emitted IRI and label was rechecked against the register.

Fields that were being dropped, now read: the contact person and the repository contact, the
owning collection, dc.identifier.other, dc.source.uri, local.demo.uri, every dc.relation
qualifier, dc.format, dc.coverage.placeName, given/family name parts, language names,
local.sponsor, local.additional.metadata, local.size.info, local.refbox.format and the metashare
tree. Bitstreams of the ORIGINAL bundle now become distributions with format, size and checksum.

Portability, none of which changed LINDAT output: ISO 639-1 and region-tagged language codes are
accepted, the stock DSpace type list is mapped, dates are proved castable before being emitted,
the funding guard names the whole grantAgreement prefix, and a repository with no url is no longer
described as living at each item's own address.

Tests: 70 methods over 9 fixtures, including a stock-DSpace record, every date shape DSpace
stores, and the two CLARIN access shapes. Element order is asserted directly - the CCMM schema is
an xs:sequence and validating against it would pull xml.xsd and GML over the network at build
time - plus closed-set guards so no off-register string can come back. Mutation testing: 45 of 46
single-edit mutants are now caught, against 20 of 72 before.

Re-run over all 2628 live records: 2628/2628 valid against CCMM 1.1.0 in XSD 1.1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
XmlMatcherBuilder builds its NamespaceContext from a plain map with no built-in prefixes, so
JAXP cannot resolve the reserved "xml" prefix and ccmmXmlLangFollowsTheXoaiLanguageWrapper
failed with "Prefix must resolve to a namespace: xml". Register it alongside ccmm.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tested

Copilot asked for coverage of the embargo/restricted access paths and of the FormatDate
variations. Most of what it listed is already asserted, and one of its premises was wrong -
the crosswalk never reads local.embargo.termslift, embargo comes from others/access-status.
Three branches genuinely had no assertion, so they get one:

- others/restrictedAccess=true on a PUB record with files must still be restricted access;
  the download gate outranks the licence category. 19 live records take that path.
- others/access-status=embargo must reach the COAR embargo term, not the restricted fallback.
- a dc.date value carrying a time must stay a ccmm:date_time; time_instant is a choice, and
  truncating it into ccmm:date would silently drop the time.

73 test methods over 11 fixtures.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
license-maven-plugin checks src/** and fails the dspace-oai build without it. The check runs in
the verify phase, so it only surfaced once the unit tests stopped failing first.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every record already validated against dataset/schema.xsd, which is why none of this showed up
as invalid. The CCMM specification states requirements the XSD has no way to express - they live
in en/dsv.ttl as scope notes and cardinality - and a conformance check reads those, not the XSD.
All 89 scope notes and 92 cardinality statements were extracted and checked against the output of
all 2628 live LINDAT records.

The one the specification calls mandatory:

- FRASCATI FORD. Dataset.hasSubject: "At least one subject must be a value from FRASCATI FORD
  vocabulary. For that subject use reference codelist .../SubjectCategory/ as the Subject scheme."
  No record carried one. The field is read off dc.type: a tool or service is computer science
  (10201), a newsreel clip is History (60101) - on this corpus every dc.type "clip" is a segment
  of a digitised 1943 newsreel held as a historical source, not a language resource - and
  everything else is Linguistics (60203), which is what the repository collects. Codes and both
  prefLabels come from the live register, and FORD_DEFAULT_CODE is the single value another
  deployment changes. 1161 History, 1136 Linguistics, 331 Computer sciences.

Statements the output was making that the source does not support:

- access_url pointed at the bitstream. accessUrl: "The resource at the access URL contains
  information about how to get the Dataset. It shall be the web page (not a document file)", and
  the schema's own Czech documentation says to use it "in all cases except a downloadable file
  URL". It is now the item's landing page; download_url keeps the file.
- identifier/value carried the resolved URL. CCMM types value as skos:notation - the identifier
  WITHIN its scheme - and keeps the resolvable form in iri, the split its own samples use. One
  emitter now serves every source, so scheme, notation and IRI cannot disagree between branches.
- The dataset had no iri of its own, so the published record had no resolvable identity.
- license is 1..1 and was serialised as <license/> on 584 records. A void mandatory element tells
  a harvester nothing; those records now carry rightsstatements.org UND, which is the standard
  statement for rights not stated.
- Language labels were DSpace's own strings, not the register's, and glued words together
  ("UpperSorbian", "AncientGreek (to 1453)") on 537 elements; their positional pairing could also
  slip once duplicate codes were removed. The scope note asks for the register's "label and/or
  its 3-letter code" and CCMM's own dataset-mini sample writes primary_language with an iri and
  no label, so the IRI alone is published.
- media_type published IANA IRIs for six types RFC 6838 reserves as unregistered (application/
  x-gzip and friends); those IRIs 404. They are suppressed, and the mandatory EU format element
  carries the file type - which is the scope note's own "otherwise Format MAY be used" branch.
- format carried the IANA media type where Format.label asks for a label from the EU file-type
  codelist. format is now the EU concept, media_type stays IANA. All 9125 distributions map to a
  real EU concept; every code was verified by content, because that register answers 200 for any
  path.

Values that were not what they claimed to be:

- 203 byte-identical sibling elements across 161 records, because DSpace stores one value under
  several language wrappers and two sources can carry the same value. Deduplication now happens
  where each block is assembled, so it also catches the cross-source repeats.
- "Germany||United States" was published as one place name; it is two.
- 20 alternate titles said nothing about how they differ from the title.
- dc.relation.uri was listed as a qualifier but had no term, so those related resources came out
  untyped; the register has Other for exactly that.
- An iri only had to start with a URL, so "http://a/ Book and enhanced publication" was published
  as an identifier. The whole value now has to be one URI.
- A packed contact field put an affiliation in the email slot.
- A second dc.title never became an alternate title. The predicate was
  doc:field[@name='value'][position() > 1], but position() counts within each parent and every
  XOAI language wrapper holds exactly one field, so it was always 1. Two titles only worked when
  both sat in the SAME wrapper, which XOAI does not do for values differing in language. Latent
  on this corpus - no record has two dc.title - but wrong, and the new fixture exercises it.

Measured after the change over all 2628 records: 2628 valid, every counter zero, 95 tests and
187 assertions green, 67 of 68 stylesheet mutants killed, 23 portability inputs clean. The
surviving mutant edits a FormatDate branch its callers make unreachable by pre-filtering.

Not changed, and why: eight records carry U+FFFD in title and description text. That is in the
source data - eight records in, eight records out - so guessing the intended character would be
inventing content. It belongs in the repository's metadata.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@milanmajchrak

milanmajchrak commented Sep 11, 2026

Copy link
Copy Markdown
Collaborator Author

ccmm-transformacie.html
Moje review:

  1. iri: wtf, co je others@handle, akoze zoberie nejaky iny handle?
  2. Je to presne naprogramovane tak, ako si popisal v tom html?
  3. Ten html dokument je dost fajn, nech ho urobi aj v en
  4. Bod 6.1. access_rights. Na zistenie ci je to open alebo nie nemozes pouzivat rights.label - musis to zistit normalne cez authorizacne requesty, ak je to pre anonymous dostupne, tak je to v poriadku. Ale este pred tym pozri ako to ma CMDI, ak to ma CMDI takto ,tak to nechaj tak.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

export new DataCite/CCMM metadata over OAI-PMH

2 participants